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(57) Abstract 

The present invention provides a method and 
system for estimating the motion within a video se- 
quence. The invention provides very accurate esti- 
mates of both the displacement vector field, as well 
as, the boundaries of moving objects. The system 
comprises a preprocessor (102), a spatially adaptive 
pixel motion estimator (104), a motion boundary es- 
timator (106), and a motion analyzer (108). The 
preprocessor (102) provides a first estimate of the 
displacement vector field, and the spatially adaptive 
pixel motion estimator (104) provides a first estimate 
of object boundaries. The motion boundary estima- 
tor (106) and the motion analyzer (108) improve the 
accuracy of the first estimates. 
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METHOD AND SYSTEM FOR ESTIMATING MOTION WITHIN A VIDEO 

SEQUENCE 



Field of the Invention 



5 



The present invention relates generally to video coding, and more particularly 
to using motion estimation in video coding. 



A video sequence consists of temporally sampled projections of a three 
dimensional, 3-D, scene onto the two dimensional, 2-D, image plane. The 3-D 
motion that occurs within this scene is captured as 2-D displacements of these 
15 projections. The displacement of a particular picture element, pixel, in the current 2- 
D image plane, may be represented by a vector which points to the location of the 
particular pixel in a previous image plane. The displacement vector field, DVF, 
describes the motion of all pixels between a given set of image planes, and therefore 
. represents the 3-D motion of objects projected onto the image plane. 



Accurate estimation of the DVF within a video sequence is crucial in many 
applications of image sequence processing. Video coding, frame interpolation, object 
tracking, and spatio-temporal motion compensated filtering are all applications that 
require accurate estimation of the DVF to utilize the interframe correlation that exists 
25 in a video sequence. 

Compression of digital video to a very low bit rate, VLBR, is a very important 
problem in the field of communications. In general, a VLBR is considered not to 
exceed 64 kilo-bits per second (Kbps) and is associated with existing personal 
30 communication systems, such as the public switch telephone network and cellular 
systems. To provide services like video on demand and video conferencing on these 
systems, would require the information contained in a digital video sequence to be 



10 



Background of the Invention 



20 
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compressed by a factor of 300 to 1 . To achieve such large compression ratios, 
requires that all redundancy present in a video sequence be removed. 

Current standards, such as H.261, MPEG1, and MPEG2 provide compression 
5 of a digital video sequence by utilizing a block mouon-compen sated Discrete Cosine 
Transform, DCT, approach. This video encoding technique removes the redundancy 
present in a video sequence by utilizing a two-step process. In the first step, a block- 
matching, BM, motion estimation and compensation algorithm estimates the motion 
that occurs between two temporally adjacent frames. The frames are then 
10 compensated for the estimated motion and compared to form a difference image. By 
taking the difference between the two temporally adjacent frames, all existing 
temporally redundancy is removed. The only information that remains is new 
information that could not be compensated for in the motion estimation and 
compensation algorithm. 

15 

In the second step, this new information is transformed into the frequency 
domain using the DCT. The DCT has the property of compacting the energy of this 
new information into a few low frequency components. Further compression of the 
video sequence is obtained by limiting the amount of high frequency information 
20 encoded. 

The majority of the compression provided by this approach to video encoding 
is obtained by the motion estimation and compensation algorithm. That is, it is much 
more efficient to transmit information regarding the motion that exists in a video 

25 sequence, as opposed to information about the intensity and color. The motion 

information is represented using vectors which point from a particular location in the 
current intensity frame to where that same location originated in the previous 
intensity frame. For BM, the locations are predetermined non -overlapping blocks of 
equal size. All pixels contained in these blocks are assumed to have the same motion. 

30 The motion vector associated with a particular block in the present frame of a video 
sequence is found by searching over a predetermined search area, in the previous 
temporally adjacent frame for a best match. This best match is generally determined 
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using the mean-squared-error (MSE) or mean -absolute -difference (MAD) between 
the two blocks. The motion vector points from the center of the block in the current 
frame, to the center of the block which provides the best match in the previous frame. 

5 Utilizing the estimated motion vectors, a copy of the previous frame is altered 

by each vector to produce a prediction of the current frame. This operation is referred 
to as motion compensation. As described above, the predicted frame is subtracted 
from the current frame to produce a difference frame which is transformed into the 
spatial frequency domain by the DCT. These spatial frequency coefficients are 
10 quantized and 

entropy encoded providing further compression of the original video sequence. Both 
the motion vectors and the DCT coefficients are transmitted to the decoder, where the 
inverse operations are performed to produce the decode video sequence. 

15 The estimation of the DVF within a video sequence is an extremely difficult 

problem. The two main sources of the difficulties in obtaining accurate estimates are 
the nonstationarity of the DVF and the ill-posed nature of the problem. The ill- 
posedness results from the violation of Hadamard's definition of a well-posed 
problem, which is characterized by the existence of a solution, the uniqueness of a 

20 solution, and the continuity of a solution. The problem of estimating the 

displacement field violates all three of these properties. Objects moving in an image 
sequence will occlude certain areas and uncover others: at these locations the DVF is 
undefined and no solution exists. Object deformation and changes in the camera's 
depth of field can also result in regions where the DVF is undefined. For a given 

25 image sequence, many displacement fields may satisfy the data and thus the solutions 
are not unique. The continuity property is also violated, since in some image 
sequences even slight modifications in the local intensity values can cause large 
changes in the magnitude and/or direction of the displacement estimate. The fact that 
the DVF estimation problem is ill-posed, must be taken into account if any useful 

30 results are to be obtained. 
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The nonstationarity of the DVF results from objects moving along different 
trajectories within the scene that causes discontinuities to occur at the object 
boundaries, as well as from regions where the motion is undefined due to covered or 
uncovered parts of the moving scene. The nonstationarity of the DVF implies that 
5 any approach used in its estimation; must be adaptable. That is, approaches which 
assume stationarity will result in estimates of the DVF that distort the boundaries 
between differently moving objects. These distortions in the motion boundaries 
result directly in distorted objects, and therefore an increase in the entropy of the 
difference image. 

10 

The assumption used by BM that the motion within a block is constant , 
effectively constrains the problem of estimating the DVF so that the problem is no 
longer ill-posed. However, this same assumption also requires that if the motion 
boundaries are to be preserved they must coincide with the block boundaries. Since 

15 in real sequences this rarely occurs, significant errors in the estimation of the DVF 
result at the boundaries. Such errors can be described as a blurring of the boundaries 
of the DVF. For example, if a block contains a moving object and stationary 
background, then depending on the size of the object either part of the stationary 
background will be displaced or the moving object will be treated as stationary 

20 background. In either case, the motion compensated block of the previous frame will 
be a poor prediction of the block in the present frame, resulting in a DFD image with 
increased entropy. 

Another approach for estimating the DVF which has recently gained in 
25 popularity is the spatio-temporal gradient approach. Spatio-temporal gradient-based 
motion estimation algorithms are derived by minimizing the displaced frame 
difference, DFD, at each pixel using a temporal gradient and spatial gradient in the 
previous frame, based on an initial estimate of the DVF. One common approach to 
carry out this minimization is that of first linearizing the DFD using a prediction of 
30 the DVF. By assuming that all the pixels in a neighborhood of the working point 
undergo the same motion, a set of linear equations results, instead of a single 
equation. If the neighborhood allows recursive computability, then algorithms 
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resulting from this approach are generally referred to as pixel recursive, PR, motion 
estimators. PR algorithms are generally faster than BM algorithms, and do not 
require spatial interpolation for subpixel accuracy, although PR algorithms may 
require spatial interpolation to compute the spatial gradients in the previous frame 
5 along the direction of motion. Furthermore, PR algorithms can be extended to handle 
more complicated types of motion than the purely translational type. 



Similar to block matching, the performance of PR algorithms also suffers at 
the boundaries separating differently moving objects. That is. at these motion 
1 0 boundaries, pixels in the local neighborhood do not undergo similar motion. By 
assuming that they do, PR algorithms produce inaccurate estimates near these 
boundaries. As in the case of block matching, the poor estimates of the DVF near 
these boundaries will result in a marked increase in the entropy of the DFD.~ 

15 There are several major problems with both block matching and spatio- 

temporal gradient approaches. For instance, the assumption that the motion within a 
block or local neighborhood is homogeneous results in severe degradation of the 
boundaries within the DVF. Block matching algorithms can not resolve within a 
single block complicated types of motion, such as rotations. Also, they require 

20 . spatial interpolation for sub-pixel accuracy. Spatio-temporal gradient approaches 
suffer from the fact that the linearization of the DFD is extremely dependent on the 
initial prediction of the DVF. These difficulties result in an increase in the DFD's 
entropy, which is prohibitive for very low bitrate coding applications. A very low 
bitrate is defined as a rate less than 64 kbits/sec. 



25 



Therefore, a need exists for a method and system for accurately estimating 
motion within a video sequence. The method and system are required to regularize 
the estimation of the displacement vector field, DVF, to combat ill-posedness and 
estimate the boundaries or discontinuities that exist within the DVF. 



30 



WO 97/04600 



PCT/US96/10141 



6 

Brief Description of the Drawings 

FIG. 1 is a diagram of a preferred embodiment of a system for estimating 
5 motion in accordance with the present invention. 

FIG. 2 is a diagram of a preferred embodiment of a preprocessor in 
accordance with the present invention. 

10 FIG. 3 is a diagram of a preferred embodiment of a spatially adaptive motion 

estimator in accordance with the present invention. 

FIG. 4 is a diagram of a preferred embodiment of a first DVF update circuit in 
accordance with the present invention. 

15 

FIG. 5 is a diagram of a preferred embodiment of a motion boundary 
estimator in accordance with the present invention. 

FIG. 6 is a diagram of a preferred embodiment of a motion analyzer in 
20 accordance with the present invention. 

FIG. 7 is a flow diagram of steps for implementing a preferred embodiment of 
a method for estimating motion in accordance with the present invention. 

25 FIG. 8 is a flow diagram of steps for implementing a preferred embodiment of 

a method for segmenting in accordance with the present invention. 

FIG. 9 is a flow diagram of steps for implementing a preferred embodiment of 
a method for adaptively estimating motion in accordance with the present invention. 

30 
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FIG. 10 is a flow diagram of steps for implementing a preferred embodiment 
of a method for estimating motion boundaries in accordance with the present 
invention. 



The present invention provides a method for obtaining very accurate estimates 
of both the displacement, as well as, the boundaries of the moving objects within a 
video sequence. Therefore, the problems of ill-posedness and nonstationarity of the 
boundaries are solved simultaneously. An estimate of the object boundaries within a 
15 video sequence is obtained through the segmentation of the present image frame into 
regions of similar intensity. The boundaries separating these regions are considered 
to be the boundaries of the objects within the sequence. This object boundary 
information is used to determine a first estimate of the D VF. 

20 Utilizing the first estimate of the DVF. a first estimate of the object 

boundaries is obtained with the accuracy of the location of each boundary improved, 
as well as, the removal of any boundary found in the estimate to be non-moving. 
Based on this estimate of the moving object boundaries, a second and improved 
estimate of the DVF is also determined . As a final step the second estimate of the 

25 DVF and the estimated moving object boundaries arc further refined based on a 
predetermined model of the DVF. Such models typically take into account 
complicated types of motion, such as object rotations in the field of view and 
apparent object motion due to changes in the camera focal length or position. The 
third estimate of the DVF and the second estimate of the moving object boundaries 

30 are the result of fitting such a model to the second estimate of the DVF and the 
moving object boundary estimate. 



5 



FIG. 1 1 is a flow diagram of steps for implementing a preferred embodiment 
of a method for analyzing motion in accordance with the Dresent invention- 



Detailed Description of the Preferred Embodiments 



10 



WO 97/04600 PCT/US96/10141 



8 

FIG. 1, numeral 100, is a diagram of a preferred embodiment of a system for 
estimating motion in accordance with the present invention. The system includes a 
preprocessor (102), a spatially adaptive motion estimator (104), a motion boundary 
estimator (106), and a motion analyzer (108). The preprocessing section (102) 
5 provides a object boundary estimate (112) of a current intensity frame (1 10), f k at 

time instant k . Based on this object boundary estimate (112), the current intensity 
frame (1 10), f k , and a previous intensity frame (114), f k _ x , the spatially adaptive 

motion estimator (104) provides a first DVF estimate (1 16), dk * The DVF 
characterizes the motion within the scene that occurs during the time separating 

10 frames f k and f h _ x . Both the first DVF estimate (116) and the object boundary 

estimate (112) are further refined in the motion boundary estimator (106) and motion 
analyzer (108). Specifically, using the first DVF estimate (116),^, the motion 
boundary estimator (106) refines the object boundary estimate (112), eliminating the 
intensity boundaries which do not correspond to motion boundaries. The output of 

15 the motion boundary estimator (106) is the first moving object boundary estimate 

(118), fa , and represents only the boundaries of objects that are moving within the 
video sequence. Also, inaccuracies in the first DVF estimate (1 16) are removed in 
the motion boundary estimator (106) resulting in the second DVF estimate (120) f J^ • 
These inaccuracies are generally due to the corruption of the input video sequence 
20 with noise. The motion analyzer (108) further refines both estimates, <4 ^ k * 

using modeling approaches based on the principles of object motion. The third DVF 
estimate (122), , and the second moving object boundary estimate (124), , 
represent extremely accurate estimates of the DVF and the moving object boundaries, 
respectively. 

25 

FIG. 2 , numeral 200, s a diagram of a preferred embodiment of a 
preprocessor in accordance with the present invention. The system includes an order 
statistics filter (202) and an object boundary estimator which is comprised of a object 
grower (206), and a object merger (206). The major function of the preprocessing 
30 step is to provide the spatially adaptive motion estimator (104) with an accurate 
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segmentation of the current intensity frame (210) into regions or objects. To achieve 
this goal, a copy of the current intensity frame f k (210) is first filtered separately 

using an order statistics filter (202). This operation is performed to remove any small 
objects from the sequence prior to segmentation. These smaller objects, which are in 
5 general the result of illuinination changes in the video sequence and not a true object, 
can degrade the spatially adaptive motion estimator's (104) ability to accurately 
estimate the motion in that object 



10 frame is a 7x7 median filter. This filter is described by the following operation 



15 

where y k (i,j) is the filtered version of f k 0J). The filtered output (212), y k (ij)> is 
the median value of f k (ij) (210) and its 49 nearest spatial neighbors. Any objects 
with a spatial support, or size, that is less than 7x7 are removed by this filter. As 
discussed in the paragraph above, these smaller objects must be removed prior to the 
20 segmentation of the image frame to avoid degradation to the accuracy of the resulting 
segmentation. 

The object boundaries contained in the filtered output (212) are estimated 
using a two-step process (204). The first step (206) is called object growing, wherein 
25 . each pixel is compared to its neighbors. Based on this comparison, the pixel is 

classified as either belonging to the same object as its neighbors, or to a new object. 
The test used to determine the classification of pixel (i,j) is given by 



A filter which is typically used to remove smaller objects from an image 



v A 0*, j) = median{f k (/ - 3, j - 3), f k (/ - 3, j - 2), 



,/ Jt (/ + 3,y + 2),/,.(i+3,7 + 3)} 



| y* (/, j) - y k (/ -mj- n)\ < T 



(1) 



30 



where m and n can take on the values of {-1,0,1 }. The threshold T is used to 
determine whether y k (i, j) is a member of the same object as y h (i - m, j - n) . If the 
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absolute value of the difference between two neighboring pixels, 

\y k (i, j) - y k (i - m, 7 - , is less than the threshold 7\ then pixel (i,j) is classified as a 

member of the same object as pixel (i-mj-n). This object is denoted as 
Obj(i -mj-n). If the difference is greater than 7\ then pixel (i j) is considered not 
5 to be a member of Obj(i - mj - «). For the case when the neighbors of (i j) have not 
all been classified or y k (Uj) is determined to not be a member of the neighboring 
objects, than pixel (i j) is classified as the initial pixel of a new object. 

The threshold T is typically fixed to a predetermined value. A drawback of 
10 this approach is that this value is dependent upon the particular image to be 
segmented. An approach which removes this dependency is described by the 
following expression 



15 



y k (ij) e Obj(i - mj - n) 



y k (Uj)<MIN + T x 

and , (2) 

y k (iJ)>MAX-T x 



where MIN and MAX are the maximum and minimum intensity values contained 
within Obj(i -m 7 i-n) and T x is again a predetermined parameter. After each pixel 
is classified as belonging to a specific object, the maximum and minimum values of 
that object are tested to determine if an adjustment is needed. The tests and 
20 adjustments are the following 

if(y k (ij) < MIN\-> MIN = % (i f /), 
and (3) 

if{y k d*J) * MAX),—* MAX = y k (ij). 

25 

This operation has the effect of providing an adjustable window for each 
object that can adapt to any particular intensity frame. Typically, the predetermined 
value for the threshold T x is 25. This method for adjusting the window, i.e. 
thresholds, is extremely effective at proving a consistent segmentation over a wide 
30 variety of video sequences. 
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The second step of the object boundary estimation process, each object is 
tested to determine whether it should be merged with a neighboring object or not 
The pixels residing along the border of an object are compared with the pixels on the 
5 borders of the neighboring objects. If the difference between these border pixels is 
small, then the objects are merged. Specifically, the following test is conducted on 
the border pixels of two neighboring objects 

if{\y k (iJ)-y k (i-mJ-n)\ $ T 2 )-+ merge(Obj(iJ)&Obj(i-nJ-m)) , 

10 (4) 

where T 2 is a predetermined parameter typically set to a value of 30. 

A second comparison is also performed in an attempt to again guard against 
15 the creation of small objects. These smaller objects are generally the result of 

gradually varying Ulumination changes which the object growing algorithm interprets 
as object boundaries. This test compares the size of an object, i.e. the number of 
pixels contained in an object, to a third predetermined threshold. If the number of 
pixels is less than this predetermined threshold, than the object is merged with a 
20 neighboring object. This merging operation is performed using the neighboring 
object which is the most similar to the current object. The amount of similarity 
between two neighboring objects is measured using the border differences described 
in Equation (4). Typically, the size threshold is set at 256 pixels. The object merging 
operation (208) is effective at removing smaller objects which may have been formed 
25 during object growing operation. 

An advantage of this two-step object boundary estimation approach (204) , as 
opposed to other segmentation approaches, is that it is guaranteed to provide 
boundaries that are continuous and closed. For the application of motion estimation 
30 and analysis this is a very important result, since we know that in general the objects 
within a video sequence are defined by a continuous and closed boundary. 
Furthermore, the boundaries contained in an intensity frame are a super-set of the 
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corresponding DVF boundaries. Therefore, the boundaries or discontinuities present 
in the DVF are also continuous, closed, and well defined. 

The output of the preprocessing unit (102) is the object boundary estimate 
5 (216) S k . The object boundary estimate (216) S k assigns each pixel in y k (ij) (212) 

a value corresponding to the object it is a member of. The boundary is retrieved from 
(216) S k by determining where the object numbering changes. The object boundary 
S k (216) combined with the present intensity frame f k (1 10) and the previous 
intensity frame (114), are used by the spatially adaptive motion estimation unit 

10 (104) to determine the first estimate of the DVF ^ (1 1 6). As discussed above, the 
object boundary estimate S k (1 12) contains the discontinuities of the DVF d k , and 

therefore, is considered as the first estimate of the boundaries between differently 
moving objects. The first or initial estimate of the line process, S h , is used as the 

mechanism by which the spatially adaptive motion estimation algorithm adapts to 
15 these discontinuities. 

FIG. 3 , numeral 300, is a diagram of a preferred embodiment of a spatially 
adaptive motion estimator in accordance with the present invention. The spatially 
adaptive motion estimation unit (104) comprises a causal look-up table (312), a DVF 

20 predictor device (318), and an update circuit (320). The inputs to the spatially 
adaptive motion estimation unit are the estimate object boundaries S k (304), a 
predetermined causal local neighborhood of past displacement estimates d k (306) , 
the previous intensity frame (308), and a current intensity frame f k (310). The 
causal look-up table device (312) stores a set of predetermined autoregressive, AR, 

25 prediction coefficients (3 1 4), and corresponding prediction uncertainties (3 1 6). 

The predetermined set of causal AR prediction coefficients (314) and 
corresponding uncertainty terms (316) are typically found empirically. Generally, a 
least squares estimation approach is used on either a prototype or previously 
30 estimated DVF. Based on the object boundary estimation S k (304) a subset of the 
predetermined AR prediction coefficients (314), a(m,n\S k ), and a corresponding 
uncertainty term, w(i,j\S k ) (316), are chosen from the look-up table (312) for use in 
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the DVF predictor (318) and update circuit (320). The DVF predictor circuit 
determines a prediction of the DVF based on the subset of AR prediction coefficients 
(3 14) and a local neighborhood of displacement vectors (306). The prediction 
operation is described by the following equation 

5 

dh (L j) = T.a(m,n\S k )d k (i - m, _/ - n) , (5 ) 

where dk(ij) is the prediction of the motion occurring at pixel location r = in 
the current image frame, and a(m,n\S k ) are the AR prediction coefficients with a 
10 local support R. 

The local neighborhood R consists of the following pixel locations: the pixel 
in the column directly to the left (i,j-l), the pixel in the row above and the column to 
the left (i-1 j-1), the pixel in the row above (i-1 j), and the pixel in the row above and 

15 the column to the right (i-1 j+1). It should be noted that the choice of R is made at 
the time of implementation and is dependent on the method used to navigate through 
the two dimensional data sets used to represent the image and displacement - - 
information at a particular time instant. For this particular R , it is assumed that the 
data is access from left to right across each row , starting with the top row. Other 

20 methods for navigating through the image and displacement data can also be used. 
This would require a slight modification to the local neighborhood R . however the 
operations would remain the same. 

As described by Equation (5) and discussed above, if the object boundary 
25 estimate (304) S k indicates that an object boundary is present in a predetermined 
local neighborhood R , the AR coefficients a(m,n\S h ) (314) are adapted so as not to 
include in the prediction any displacement vectors belonging to another object. The 
non-stationary assumption on which the above equation is based, is valid throughout 
the DVF and results in consistently accurate predictions. The inaccuracies incurred 
30 by a stationary model due to the mixing or blurring of displacement vectors located 
near the object boundaries, is alleviated. 
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The predicted displacement vector d k (ij) (324), the associated uncertainty 
term w(i 9 fiS k ) (316), the previous intensity frame f h _ x (308), and current intensity 
frame f k (3 10) are inputs to the first DVF update circuit (320). The first DVF update 
5 circuit updates the predicted displacement vector d k (/, j) (324) resulting in the first 
DVF estimate (322). 

FIG. 4, numeral 400, is a diagram of a preferred embodiment of a first DVF 
update determiner in accordance with the present invention. The first update circuit 
10 comprises a motion compensation unit (402) and a gain calculator unit (404). The 
motion compensation operation performed by the motion compensation unit (402) is 
a nonlinear operation which is described by the following equation 

/*(0 = / Jt _ I (r-3 jt (r)), (2) 

15 

where f k (r) (422) is the motion compensated previous frame. The value 

(r - d k (r)) (422) is found by taking the intensity value of the pixel located 
spatially at r - d k (r ) in the previous frame (408). The motion compensated 
value f k (r) (422) is subtracted from the value of the pixel at location (r) in the 
20 current frame f k (r) (412) resulting in the displaced frame difference, DFD, signal 
(424), e k (Lj) . The DFD (424) signal is an error signal which is related to the 
accuracy of the predicted displacement vector d k (ij) . The DFD signal e k (ij) 
(424) is multiplied by a gain term (426) and added to the DVF prediction (406) 
resulting in the first estimate DVF estimate (41 4) . 

>5 

The gain determined by the gain calculator unit (404) is used to scale the DFD 
signal e k (r) (424) prior to updating the DVF prediction (406) 3*(r) . The expression 
used to calculate the gain is found by linearizing the motion compensated frame 
about the spatial location (r-2*00-*(r)), where u(r) is the error in the prediction 
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dk (r) , (406) and solving for the best linear estimate of u(r ) . The resulting 
expression for the gain, K{iJ), (426) is given by 

K{Uj) = [G t (/,,*)£(/, j) + w(/\ j)]' 1 G(i, j) 

where G(i,j) is a nx2 matrix of spatial gradients evaluated in the motion 
compensated previous frame (422) for each pixel in the local neighborhood R . 
Specifically, for the local neighborhood R described above, G(i,j) is given by 



10 



G(iJ) = 



V r f k {ij) 
V r /,0\y-D 
V T ~f k (i-lJ-\) 

V r / I .(/-1.) + 1) 



15 



where 



v 7 /<(/,;) = 



£ 

dx 
df_ 

dv 



is the two dimensional spatial gradient evaluated at pixel location (i,j) in the motion 
compensated previous frame (422) f k -\{r-dk(r)). 



As mentioned above, the gain K(U j) (426) determined by the gain calculator 
20 is used to the scale the DFD signal (424). This scaled DFD signal is added to the 
predicted displacement vector (406), dk (;, j) , resulting in the first estimate of the 
DVF(414),3t (i,;). The operation used to update dt(Lj) is described in more 
detail in the following equation 
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dk (i 9 j) = d* (/, j) + K(i y j) . E(U j) , 

where E(U j) is the DFD evaluated for each pixel in the local neighborhood R . 

5 FIG. 5, numeral 500, is a diagram of a preferred embodiment of a motion 

boundary estimator in accordance with the present invention. The motion boundary 
estimation unit (106) comprises a noncausal look-up table (510), a moving object 
boundary estimator (506), a DVF estimator (508), a noncausal local neighborhood of 
the DVF (502), and a moving object boundary estimator (506). The inputs to the 

10 moving boundary estimator are the object boundary estimate S k (526), the first DVF 
estimate d k (524), the previous intensity frame f k _ x (528), and a current intensity 
frame/* (530). The moving object boundary unit (504) stores the current estimate of 
the moving object boundary for possible use in an iterative solution. Specifically, the 
solution provided by the moving object boundary estimator (506) can be used as an 

15 initial condition for an improved solution. A method for carrying out this iteration is 
discussed below. The moving object boundary estimator (506) is initialized with the 
object boundary estimate (526). 

The moving object boundary estimator (506) provides an estimate of the 
20 moving object boundaries l k (546) based on the object boundary S k (526) and the 
corresponding first estimate of the DVF (524). The moving object boundary 
estimator (506) is comprised of a moving edge determiner unit (514), edge continuity 
unit (5 1 6), and a moving object update unit (5 1 8). The estimate of the moving object 
boundaries, l k (546), is determined by adjusting the first estimate S k . These 
25 adjustments, which are based on known statistical properties of these boundaries and 
characterized through the use of confidence measures, result in the removal of 
boundaries that do not belong to a moving object. The particular confidence 
measures used are described in detail below. As mentioned above, the moving object 
boundary estimator (506) is initialized with the object boundary estimate (526) . 
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The moving edge determiner unit (514) evaluates whether or not pixel (i,j) is 
part of a moving object boundary and assigns a confidence measure to the current 
estimate l k (ij). This evaluation is performed by comparing adjacent displacement 
vectors. More specifically, the evaluation and corresponding confidence measure 
5 (540) are given by 

Z>(w) = (rf t (i\^ 

If confidence measure D(i,j) (540) is large, then a moving object boundary is likely 
to exist. Conversely, if D(i f j) (540) is small, than the likelihood is small that a 
1 0 moving object boundary exists. 

Similar to the moving edge determiner unit (5 14), the edge continuity unit 
(516) also assigns a confidence measure (542) to the current estimate of the moving 
object boundary at pixel (i j). This confidence measurement (542) however, is a 

15 function of the values of the neighboring boundary elements. Based on the 

assumption that objects within a video scene have boundaries that lie on a closed 
continuous curve, a pixel (i,j) is a boundary element, if and only if, it lies on such a 
curve. In other words, if pixel (i,j) is determined to be a moving boundary element, 
than the adjacent pixels in a particular direction most also be boundary elements. The 

20 above characterization is captured in the following expression 

C(i, j) = 0 .5(/(/, 7)+ /(* - 1, j) + /(/, j - 1)) . 

25 The moving object update determiner adjusts the moving object boundary 

estimate /*(/,./), at pixel (ij), based on the current estimate of the moving object 
boundary, which is initially the object boundary estimate (526) S k (iJ), and the 
confidence measure D(iJ) (5 4)and C(iJ) (542). The expression which 
characterizes the moving object boundary update determiner is given by 

30 
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1 



l + exp(L(/,y)) 
where 

L{U j) = p(a[(l - £ • C{U j)) + (1 - S h (/, j)))- X*D(i. j)) , 

5 

The coefficients a and A are predetermined parameters used to control the amount 
that each confidence measure can influence the estimate l k . The parameter p (556) 
is used to control the response of the update determiner. If p is small, then the effect 
of the confidence measures on the adjusted l k (ij) is also small. If on the other hand 

10 p is large, then the adjusted l k (ij) is dominated by the confidence measure. 

Typically, the estimation of l k and d k is done iteratively, with p being initialized to 
a small value and increased slowly at the beginning of each new iteration. The 
decision to increase p and to refine the estimates of l k (ij) and d k (ij) is made in 
the estimation terminator (516). The estimation termination circuit (514) is discussed 

15 in more detail below. 



The DVF estimation determiner (508) is comprised of an advanced prediction 
determiner (520) and a second DVF update determiner (522). Similar to the DVF 
predictor (318) described above, the advanced predictor (520) provides a prediction 
of the DVF based on the moving object boundary estimate (536). Utilizing this 
moving object boundary estimate (546) 7* (/,;), a subset of the predetermined 
noncausal AR prediction coefficients, b(m,n\l k ) (548), and a corresponding 
uncertainty term , v(i tJ ll k ) (552), are chosen from the non-causal look-up table (510) 
for use in the advanced predictor (520) and second DVF update determiner (522). 
The advanced predictor (520) determines a prediction of the DVF based on the subset 
of AR prediction coefficients (548) and a local neighborhood of displacement vectors 
9t . It should be noted that the local neighborhood 91 is different from the R used by 
the DVF predictor (318). This difference comes from the fact that a complete 
estimate of the DVF already exists. Therefore, the dependence on the scan that exists 
for the DVF predictor is not an issue for the advanced predictor. In other words, the 
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restriction to use a neighborhood that contains only previously estimated 
displacement vectors is no longer necessary. The local neighborhood 9t includes the 
nearest neighbor pixels surrounding (i j). Specifically, 91 includes the following 
pixels, (ij-1), (i j+l), (i-1 j), (i+1 J). In addition to the model described by Equation 
5 (5) the advanced predictor also utilizes a rigid body assumption to further constrain 
the prediction- The rigid body assumption specifies that all pixels within an object 
undergo similar motion . 

The advantage of combining this additional constraint with the AR model 
10 described in Equation. (5), is that a probabilistic characterization of the DyF can be 
derived in the form of an A Posteriori probability density function. The advanced 
prediction unit determines a prediction of the displacement vector d k 0J) that 

maximizes the A Posteriori function. Typically, this maximization process is 
performed using the following iterative algorithm 



15 dr\ij) = %(i.j)-* 



t 1 s [d?(ij)-k 2 ii(U)] 

A 2 1- Xfl(/n.«l /,(/,;)) 

\ m.n&R 



where 

A = -//[£>(/, - /(i, j)) + D(i + Ky)(l - /( / + 1, ;*))], 
t? is the correction step size, i.e., it is equivalent to a steepest descent minimization 
step, and n is the iteration number. The initial condition used to begin the iteration is 
20 dl = dt.. Since the first DVF estimate d k is of high quality, the improvement 
resulting from iterating is small. Generally, the number of iterations performed is 
fixed to 1. However, a criterion that monitors the percent change between iterations 
can also be used to terminate the iteration. More specifically, if 

Z\dr\iJ)~dZ(iJ)( 
25 hi , xl00< 7 
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the iteration is terminated. Typically, the threshold Tis set to 10" 3 . Again due to the 
quality of the first DVF estimate the correction step size is fixed to a value of 
0.1. 



updated by the second DVF update determiner (522) to produce the second DVF 
estimate d k (534). The second DVF update determiner (522) the same update 
method as described for the first DVF update circuit (320). 

10 As discussed above, the coupled solution for d k (534) and l k (532) is 

typically solved using an iterative process. That is, steps used to obtain both l k and 
d k are repeated using the previous results as the initial conditions. Furthermore, is 
increased after each iteration providing more weight to the confidence measures. By 
iterating the overall accuracy of both l k and d k is improved. The decision whether to 

15 perform another iteration or not, is determined in the iteration terminator (516). The 
iteration terminator utilizes a termination criteria similar the one used by the second 
DVF estimator (510). That is f the number of iterations set to a fixed number or it can 
determined based on the percent change between iterations. Typically, a maximum 
number of 3 iterations is set. However, if the percent change between iterations is 

20 below a threshold the iteration is terminated. The specific, if the following is true 



5 



The prediction d k (ij) provided by the advanced prediction unit (520) is 



li\dr(iJ)-dZ(ijf 




xl00< T 



and 



25 




x 100 < T, 
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the iteration is terminated. 

FIG. 6, numeral 600, is a diagram of a preferred embodiment of a motion 
analyzer in accordance with the present invention. The motion analyzer which 
5 provides a parameterized representation of the DVF using d k and l k as inputs. The 
motion analyzer comprises a memory device (614), translational motion estimator 
(604), zoom estimator (606), rotation estimator (608), and a object labeler and center 
determiner (612). The second estimates of the DVF d k and line process 7* are the 
inputs to the motion analyzer unit. Based on these inputs and a four parameter 
10 motion model, a third and final set of estimates of the DVF and line process is 
determined. 

The four parameter motion model characterizes the DVF by utilizing, as rigid 
bodies, the objects defined by the line process. Specifically, the displacement of each 
15 pixel contained within a particular object is characterized by the following the 
expression 

M'J) = h( i JlObj(iJ)) + Z(Obj(iJ)'cp(iJ\Obj(iJ))+e(0^^ 

20 where t k (i,flObj(iJ)) = [r x AiJ^Obj(iJ)U y t (i tJ WbMj))J is a vector representing 
the translational motion component of d k (ij), while Z(Obj(ij) and 0{Obj(iJ)) 
represent the zoom and rotation components. It is important to note that each of these 
four parameters is dependent upon the object, Obj(iJ), that a particular pixel (i j) is 
assigned. Furthermore, the term cp(ij\0bj(ij)) represents the distance from the 

25 center of Obj(iJ) to the pixel (ij). The position of each object's center is located by 
the object center determiner (612). Based on the estimate l k , the center of each object 
is determined to be the intersection of the two lines which contain the maximum and 
minimum horizontal and vertical pixel locations in Obj(iJ). 



The translational motion estimator determines the translational motion 
component for each object. This is accomplished by averaging the horizontal and 
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vertical displacement components over each object. Specifically, the translational 
components t xJt and t y k are calculated using the following expressions 



t x AU\Obj{iJ)) = ±- -£d xk (ij) 

" ijzRtgion(iJ) 

5 and 

where N is the total number of pixels contained within ObjQJ). 

The zoom estimator (606) estimates the zoom parameter Z(Obj(ij)). This 
10 parameter is used to characterized any changes that may occur in an object's focal 
length. These changes may be caused by the camera or by the motion of the object 
closer to or away from the camera location. Utilizing the dense DVF, d k (ij), the 
zoom parameter Z(Obj(ij)) is estimated for object Obj(iJ) using the following 
expression 

15 

Z(iJ\ObMj)) = £ £r-'(,\;) . / (/ - cx(ij)) . « k0 j) - r x k (ij\Obj(ij))) 
+(j-cyVJ)) ■ (d yJt U,j) - r y . t (/, JiObj(iJ)))^ 

20 where cx(i, j) and cy(i, j) represent the horizontal and vertical indices of the center 
for the object which includes pixel (i j) mATXUj) = (/ - cx{i,j)f + (j - cy(i,j)) 2 . 

The rotation estimator (608) estimates the rotation parameter 6(Obj(iJ)) for 
each object. The rotation parameter is used to characterize any rotation that may 
25 occur in an object. These rotations are generally the result of object motion and are 
estimated in a manner similar to that used to determine Z(ObXi,j)). Specifically, the 
rotation parameter 6(Obj(i, j)) is estimated using the following expression 
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0{i,flObj(ij)) = ± ^r' l (ijy\ -0*-o</,7))-(^ 

" i.jeObj(i.j) I 

5 As mentioned above, this four parameter representation of the DVF is very 

useful in applications that require object manipulation. 

As a final step, the parameterized representation of the DVF is used to refine 
estimate of the line process l k . Based on the four motion parameters, each object is 
10 compared with its neighboring objects. If an object is determined to be similar to a 
neighboring region they are merged to form a single region. Two regions are 
determine to be similar if the following four conditions are met: 

Condition 1. 

15 (',jb(MRe*fcn(i\^)-^ < /> , 

Condition 2. 

(t y k (i, 7I Re gion{U j)) - t yJk (' + j + rn\ Re gion(i + //, j + m))f < P 2 , 
Condition 3. 

(Z(i,jlRegion(i 9 fty-Z(i + nJ + m\Region(i + nJ + m)))£P 3 
20 Condition 4. 

{6(i,j\Region(iJ))- 6(i + nj + m\Rcxian(i + nj + m)))< P 4 

where (i+n,j-fm) indicates the location of the neighboring region Obj(i + nj + m) . 

FIG. 7, numeral 700 is a flow diagram of steps for implementing a preferred 
25 embodiment of a method for estimating motion in accordance with the present 

invention. The current intensity frame is segmented (702) using a preprocessor to 
determine the boundaries of the objects contained in the scene captured by the video 
sequence. These boundaries are used by the spatially adaptive motion estimator 
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(704) to adapt to objects that are moving differently in the video sequence. A first 
estimate of the DVF is provided by the spatially adaptive motion estimator (704). 
Based on the estimator of the object boundaries and the first estimate of the DVF, a 
first estimate of the moving object boundaries and a second estimate of the DVF are 
5 obtained (706). Analyzing and modelling the first estimate of the moving object 
boundaries and the second estimate of the DVF provides a third estimate of the DVF 
and a second estimate of the moving object boundaries. 

FIG. 8, numeral 800 t is a flow diagram of steps for implementing a preferred 
1 0 embodiment of a method for segmenting in accordance with the present invention. 
The first step is to remove noise from the image frame by filtering using an order 
statistics filter (802). Next pixels are grouped into objects using a region grower 
(804). The final step includes merging small objects into larger ones using a region 
merger and a predetermined comparison test (806). 

15 

FIG. 9, numeral 900, is a flow diagram of steps for implementing a preferred 
embodiment of a method for adaptively estimating motion in accordance with the 
present invention. First, based on the object boundary estimate, a causal look-up 
table is accessed to provide at least one causal prediction coefficient and a causal 

20 uncertainty parameter (902). Next, from a memory device containing previously 

estimated displacement vectors, a causal local neighborhood is set to a predetermined 
initialization (904). A prediction of the current displacement vector is determined 
next, based on the previously estimated displacement vectors and at least one causal 
prediction coefficient (906). The predicted displacement vector is updated to become 

25 the first displacement estimate and is stored in the causal local neighborhood (908). 
The process is repeated for the entire current intensity frame (910) resulting in the 
first estimate of the DVF. 

FIG. 10, numeral 1000, is a flow diagram of steps for implementing a 
30 preferred embodiment of a method for estimating motion boundaries in accordance 
with the present invention. The first estimate of the DVF and the previously 
estimated displacement vectors of the second estimated DVF are stored in the 
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noncausal local neighborhood memory (1002). The previously estimated object 
boundaries are stored in the moving object boundary determiner (1004). Utilizing the 
object boundaries and the previously estimated displacement vectors a first estimate 
of the moving object boundaries is determined (1006). Based on the first estimate of 
5 the moving object boundaries, a noncausal look-up table is accessed to provide at 
least one noncausal prediction coefficient and a noncausal uncertainty parameter . 
(1008). A second estimate of the current displacement vector is determined next, 
based on the previously estimated displacement vectors , at least one noncausal 
prediction coefficient and the DVF update determiner (1010). This estimation 
10 process is repeated until all pixels in the current intensity frame have a displacement 
vector associated with them (1012). The process is terminated when the percent total 
change in the second estimate of the DVF and the moving boundary estimate are 
below a set of thresholds (1014). 



15 FIG. 1 1. numeral 1 100, is a flow diagram of steps for implementing a 

preferred embodiment of a method for analyzing motion in accordance with the 
present invention. First a center for each object is determined (1 102). Next, an 
estimate of the translational motion is determined for each object (1 104). This is 
followed by the estimation of a zoom parameter for each object (1 106). An estimate 

20 of a rotation parameter is also determined for each object (1 108). Finally, the 

translation, rotation, and zoom parameters are used in fitting a model to the estimated 
DVF (11 10). 



25 



The system described in FIGs. 1 through 6 may be implemented in various 
embodiments, such as Application Specific Integrated Circuit, ASIC, Digital Signal 
Processor, DSP, Gate Array, GA, and any tangible medium of/for a computer. 
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26 
CLAIMS 

We claim: 

1 . A system for estimating motion, comprising: 

A) a preprocessor, operably coupled to receive a current intensity frame 
for segmenting the current intensity frame to provide a first object boundary estimate; 

B) a spatially adaptive motion estimator, operably coupled to the 
preprocessor and operably coupled to receive the current intensity frame and a 
previous intensity frame, for providing a first displacement vector field estimate a 
first DVF estimate; and 

C) a motion boundary estimator operably coupled to the spatially 
adaptive motion estimator and preprocessor and operably coupled to receive the 
current intensity frame and the previous intensity frame, for providing a second 
displacement vector field estimate, a second DVF estimate, and a first moving object 
boundary estimate. 
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2. The system for estimating motion according to claim 1, wherein: 

the preprocessor includes: 

an order statistics filter, coupled to receive the current intensity frame, 
5 for removing noise and small objects from the current intensity frame; and 

an object boundary estimator, coupled to the order statistics filter, for 
providing the object boundary estimate; 

the spatially adaptive motion estimator includes: 
10 a causal look-up table, coupled to receive the object boundary 

estimate, for providing at least one causal prediction coefficient and a causal 
uncertainty parameter based on the object boundary estimate; 

a causal local neighborhood for storing previously estimated DVFs, 
the causal local neighborhood is set to a predetermined initialization before DVF 
15 estimation; 

a DVF predictor, coupled to the causal look-up table and the- causal 
local neighborhood, for providing a DVF prediction based on the previously 
estimated DVFs and at least one causal prediction coefficient; and 

a first DVF update determiner, coupled to receive the DVF prediction, 
20 the causal uncertainty parameter, the current intensity frame and the previous 

intensity frame, for providing the first DVF estimate, the first DVF estimate is an 
input to the causal local neighborhood where it is stored as a previously estimated 
DVF; 

25 the motion boundary estimator includes: 

a noncausal local neighborhood for storing previously estimated 
DVFs, the noncausal local neighborhood is initialized by the first DVF estimate; 

a moving object boundary unit for storing previously estimated 
moving object boundaries, the moving object boundary unit is initialized by the 
30 object boundary estimate; 
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25 



a moving object estimator, coupled to the noncausal local 
neighborhood and the moving object boundary unit, for providing a first moving 
object boundary estimate; 

a noncausal look-up table, coupled to the moving object estimator, for 
providing at least one noncausal prediction coefficient and a noncausal uncertainty 
parameter based on the first moving object boundary estimate; 

a DVF estimator, coupled to the moving object estimator and the 
noncausal look-up table, for providing the second DVF estimate; and 

an estimation terminator coupled to the moving object estimator and 
the DVF estimator, for passing the first moving object boundary estimate and the 
second DVF estimate upon termination; 

wherein the first moving object boundary estimate is an input to the 
moving object boundary unit and the second DVF estimate is an input to the 
noncausal local neighborhood; 

and where further selected, wherein: 



the object boundary estimator in the preprocessor includes a region grower 
that groups pixels into regions based on an adjustable window and a region merger 
20 that merges small regions based on a predetermined comparison test. 

3. The system for estimating motion according to claim 1, further comprising: 



D) a motion analyzer, operably coupled to the motion boundary estimator, 
for providing a third DVF estimate and a second moving object boundary estimate 
based on the second DVF estimate and the first moving object boundary estimate: 

and where further selected, wherein: 

30 the motion analyzer incudes: 

a center determiner, coupled to receive the first moving object 
boundary estimate, for providing a region location and a center point; 
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a translational motion estimator, coupled to receive the second DVF 
estimate and the center point, for providing a translational motion estimate; 

a zoom estimator, coupled to receive the center point, the region 
location, and the translational motion estimate, for providing a zoom estimate; 
5 a rotation estimator, coupled to receive the center point the region 

location, and the translational motion estimate- for providing a rotation estimate; and 

a model fitting unit, coupled to the translational motion estimator, the 
zoom estimator, and the rotation estimator, for providing the third DVF estimate and 
the second moving object boundary estimate based on the translational motion 
10 estimate, the zoom estimate, and the rotation estimate. 

4. The system of claim 1 wherein the system is embodied in a tangible medium 
of/for a computer, and where selected, one of 4A-4B: ^ 

15 4A) wherein the tangible medium is a computer diskette; and 

4B) wherein the tangible medium is a memory unit of the computer. 

5. The system of claim 1 wherein one of 5A-5C: 
5 A) the system is embodied in a Digital Signal Processor, DSP; 



20 



5B) the system is embodied in an Application Specific Integrated Circuit, 
ASIC; and 

25 

5C) the system is embodied in a gate array. 
6. A method for estimating motion, comprising: 



30 segmenting, using a preprocessor, a current intensity frame to provide an 

object boundary estimate; 
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30 

adaptively estimating the motion from a previous intensity frame to the 
current intensity frame to provide a first displacement vector field estimate, a first 
DVF estimate; and 

estimating motion boundaries based on the object boundary estimate and the 
first DVF estimate to provide a second displacement vector field estimate, a second 
DVF estimate, and a first moving object boundary estimate. 

7. The method for estimating motion according to claim 6, wherein: 

segmenting includes: 

filtering, using an order statistics filter, the current intensity frame to 
remove noise and small objects from the current intensity frame; 

region growing to group pixels into regions based on an adjustable 

15 window; and 

region merging to merge small regions based on a predetermined 
comparison test; 

adaptively estimating the motion includes: 
20 accessing a causal look-up table to provide at least one causal 

prediction coefficient and a causal uncertainty parameter based on the object 
boundary estimate; 

storing previously estimated DVFs in a causal local neighborhood, the 
causal local neighborhood is set to a predetermined initialization before DVF 
25 estimation; 

predicting a DVF based on the previously estimated DVFs and at least 
one causal prediction coefficient: and 

updating the predicted DVF to provide the first DVF estimate, the first 
DVF estimate is an input to the causal local neighborhood where it is stored as a 
30 previously estimated DVF; and 



estimating motion boundaries includes: 



WO 97/04600 



PCT/US96/10141 



31 



storing previously estimated DVFs in a noncausal local neighborhood, 
the noncausal local neighborhood is initialized by the first DVF estimate; 

storing previously estimated moving object boundaries in a moving 
object boundary unit, the moving object boundary unit is initialized by the object 
5 boundary estimate; 



estimate; 

accessing a noncausal look-up tabic to provide at least one noncausal 
prediction coefficient and a noncausal uncertainty parameter based on the first 
10 moving object boundary estimate; and 



estimated, and passing the first moving object boundary estimate and the second DVF 
estimate upon termination. 



8. The method for estimating motion according to claim 6, further comprising: 

analyzing motion to provide a third DVF estimate and a second moving object 
boundary estimate based on the second DVF estimate and the first moving object 
20 boundary estimate; 

and where further selected, wherein: 




moving objects to provide a first moving object boundary 



estimating a DVF to provide the second DVF estimate; 
terminating the estimating step when an entire frame has been 



15 



25 



analyzing motion incudes: 

determining a center to provide a region location and a center point; 
estimating translation al motion to provide a translation al motion 



estimate; 



30 



estimating zoom to provide a zoom estimate; 

estimating rotation to provide a rotation estimate; and 

fitting the translational motion estimate, the zoom estimate, and the 



rotation estimate to a predetermined model. 
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9. The method of claim 6 wherein the steps of the method are embodied in a 
tangible medium off or a computer, and where further selected one of 9A-9B: 

9A) wherein the tangible medium is a computer diskette; and 

5 

9B) wherein the tangible medium is a memory unit of the computer. 

10. The method of claim 6 wherein one of 1 OA- IOC: 

10 10A ) steps of the method are embodied in a tangible medium of/for a 

Digital Signal Processor, DSP; 

10B) the steps of the method are embodied in a tangible medium of/for an 
Application Specific Integrated Circuit, ASIC; and 

15 

IOC) the steps of the method are embodied in a tangible medium of/for a 
gate array . 
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